Using image and video compression technology, still images and motion pictures can be recorded using only a fraction of the storage required for the uncompressed video or images. A DVD, for example, is capable of holding 4.7 gigabytes of data on each layer, enough information typically for a 133 minute movie on each layer. This medium uses the MPEG-2 data compression standard that enables video programming to be efficiently stored and reproduced without significant visible degradation. The MPEG-2 compression algorithms, produced by the Motion Picture Experts Group, are described in ISO/TEC 11172 (MPEG-1) and ISO/TEC 13818 (MPEG-2), incorporated herein by reference.
MPEG-2 provides for video frames to be encoded into a series of macroblocks, with each macroblock corresponding to a different spatial portion (a 16.times.16 array of pixels) of a video image. Each macroblock includes a plurality of luminance blocks, e.g., four luminance blocks and a plurality of chrominance blocks, with each encoded block using a discrete cosine transform encoding operation. For example, a so-called 4:2:2 macroblock structure is one in which four luminance blocks and four chrominance blocks are represented.
The MPEG-2 standard defines three types of picture frames: intra, predicted and bi-directional. Intra frames, or I-frames, are coded using only information present in the picture itself. They form a reference or base frame for a series of pictures. Predicted frames, or P-frames, are coded with respect to the nearest previous I- or P-frame. P-frames use motion compensation to provide compression because only changes to the frame, and not the static background, is coded. P-frames contain both intra-macroblocks, that use only self-contained information and inter-macroblocks that use information based on predicted changes to the image. Bi-directional frames, or B-frames, use both past and a future frame as a reference. A scene having a stationary background with an object moving across it will be coded as a single I-frame representing the static background (base image), together with P- and B-frames depending on the amount of compression desired, representing the object in motion.
Interactive multimedia environments require display of both dynamic computer-generated graphics and video streams. One example is in prerecorded media, such as DVDs, where an application will generate graphic images to facilitate user interaction.
Pixel-based graphics is predicated on bit mapped images in which a graphic display is produced on a screen by reading the contents of a screen memory in which bits are stored at memory addresses corresponding to pixel screen locations. The format of pixel-based graphics is system dependent. Graphics designed for an environment having a particular specification may not be operated compatibly in a different environment.
FIG. 1 shows a typical arrangement 100 combining traditional pixel-based computer graphics with MPEG-2 based video, enabling display of both graphics and video streams. On-screen display (OSD) graphics subsystem 102 and video display controller 108 represent components of a conventional multimedia system. The OSD 102 is responsible for creating pixel-based graphics, such as button and text that may be part of a menu display. Each pixel is represented by a pattern of bits that define the color of the pixel. MPEG-2 decoder 104 converts compressed MPEG-2 data into images appropriate for viewing on a display, such as a television screen, and converter 106 converts the television ready data into an equivalent bit representation. The controller 108 creates the proper video signals, timing of image display and overlay of bit mapped data over the MPEG-2 images, and sends the output signal to the display 110.
This conventional technology, although effective for the purpose intended, combines video image decompression and computer graphics generation, which inherently are disparate technologies. Both OSD and -2 specific hardware and software are utilized, and consequently there is little sharing of resources or functionality.
Furthermore, the need to enable a user to play back material from different, possibly incompatible, playback units is hampered by pixel-based graphics. For example, a unit having a display based on 4-color graphics cannot faithfully reproduce the content of a pixel-based signal encoded in 256-color graphics. The MPEG-2 standard, on the other hand is universal and independent of other specifications of the playback unit. The invention advantageously eliminates the need for pixel-based graphics, and hence not only reduces hardware and software overhead, but maintains conformance with MPEG-2 standards. The invention additionally permits integration of graphics with video, locally and on the fly by the user of an interactive video system, subsequent to, but not limited to many attributes of, the authoring process. The invention still further enables graphics creation and display to be carried out by playback units of limited processing capabilities.
Although the best mode of the invention is described in the environment of an MPEG-2 standard, it is to be understood that the invention is applicable to other image and video compression standards, such as, but not limited to, MPEG-1, MPEG-4, JPEG, JPEG motion, any DCT based arrangement, MOV, AVI, wavelet, FLI and Adobe Flik.